Methods and systems for modeling complex taxonomies with natural language understanding

ABSTRACT

Systems and methods are presented for the automatic placement of rules applied to topics in a logical hierarchy when conducting natural language processing. In some embodiments, a method includes: accessing, at a child node in a logical hierarchy, at least one rule associated with the child node; identifying a percolation criterion associated with a parent node to the child node, said percolation criterion indicating that the at least one rule associated with the child node is to be associated also with the parent node; associating the at least one rule with the parent node such that the at least one rule defines a second factor for determining whether the document is to also be classified into the parent node; accessing the document for natural language processing; and determining whether the document is to be classified into the parent node or the child node based on the at least one rule.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/197,190, filed Nov. 20, 2018, titled “METHODS AND SYSTEMS FORMODELING COMPLEX TAXONOMIES WITH NATURAL LANGUAGE UNDERSTANDING,” whichis a continuation of U.S. patent application Ser. No. 15/294,156, filedOct. 14, 2016, titled “METHODS AND SYSTEMS FOR MODELING COMPLEXTAXONOMIES WITH NATURAL LANGUAGE UNDERSTANDING,” which is a continuationof U.S. patent application Ser. No. 14/964,511, filed Dec. 9, 2015,titled “METHODS AND SYSTEMS FOR MODELING COMPLEX TAXONOMIES WITH NATURALLANGUAGE UNDERSTANDING,” and claims the benefits of U.S. ProvisionalApplication 62/089,736, filed Dec. 9, 2014, and titled, “METHODS ANDSYSTEMS FOR ANNOTATING NATURAL LANGUAGE PROCESSING”; U.S. ProvisionalApplication 62/089,742, filed Dec. 9, 2014, and titled, “METHODS ANDSYSTEMS FOR IMPROVING MACHINE PERFORMANCE IN NATURAL LANGUAGEPROCESSING”; U.S. Provisional Application 62/089,745, filed Dec. 9,2014, and titled, “METHODS AND SYSTEMS FOR IMPROVING FUNCTIONALITY INNATURAL LANGUAGE PROCESSING”; and U.S. Provisional Application62/089,747, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FORSUPPORTING NATURAL LANGUAGE PROCESSING,” the disclosures of which areincorporated herein in their entireties and for all purposes.

This application is also related to U.S. patent application Ser. No.14/964,517, filed Dec. 9, 2015, and titled “METHODS FOR GENERATINGNATURAL LANGUAGE PROCESSING SYSTEMS”; U.S. patent application Ser. No.14/964,518, filed Dec. 9, 2015, and titled “ARCHITECTURES FOR NATURALLANGUAGE PROCESSING”; U.S. patent application Ser. No. 14/964,520, filedDec. 9, 2015, and titled “OPTIMIZATION TECHNIQUES FOR ARTIFICIALINTELLIGENCE”; U.S. patent application Ser. No. 14/964,522, filed Dec.9, 2015, and titled “GRAPHICAL SYSTEMS AND METHODS FOR HUMAN-IN-THE-LOOPMACHINE INTELLIGENCE”; U.S. patent application Ser. No. 14/964,510,filed Dec. 9, 2015, and titled “METHODS AND SYSTEMS FOR IMPROVINGMACHINE LEARNING PERFORMANCE”; U.S. patent application Ser. No.14/964,512, filed Dec. 9, 2015, and titled “AN INTELLIGENT SYSTEM THATDYNAMICALLY IMPROVES ITS KNOWLEDGE AND CODE-BASE FOR NATURAL LANGUAGEUNDERSTANDING”; U.S. patent application Ser. No. 14/964,525, filed Dec.9, 2015, and titled “METHODS AND SYSTEMS FOR LANGUAGE-AGNOSTIC MACHINELEARNING IN NATURAL LANGUAGE PROCESSING USING FEATURE EXTRACTION”; U.S.patent application Ser. No. 14/964,526, filed Dec. 9, 2015, and titled“METHODS AND SYSTEMS FOR PROVIDING UNIVERSAL PORTABILITY IN MACHINELEARNING”; and U.S. patent application Ser. No. 14/964,528, filed Dec.9, 2015, and titled “TECHNIQUES FOR COMBINING HUMAN AND MACHINE LEARNINGIN NATURAL LANGUAGE PROCESSING,” the entire contents and substance ofall of which are hereby incorporated in total by reference in theirentireties and for all purposes.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to processingdata. In some example embodiments, the present disclosures relate tosystems and methods for providing percolation of logical hierarchies innatural language processing.

BRIEF SUMMARY

In some embodiments, methods and systems for providing rules percolationof logical hierarchies in natural language processing are presented. Insome embodiments, a method may include: accessing, at a first node in alogical hierarchy configured to guide classification of a plurality ofdocuments by the natural language model, at least one rule associatedwith the first node, said at least one rule defining a first factor fordetermining whether a document among the plurality of documents is to beclassified into the first node; identifying a percolation criterionassociated with a second node in the logical hierarchy that is a parentnode to the first node, said percolation criterion indicating that theat least one rule associated with the first node is to be associatedalso with the second node; based on the identified percolationcriterion, associating the at least one rule with the second node suchthat the at least one rule defines a second factor for determiningwhether the document is to also be classified into the second node;accessing the document for natural language processing; and classifyingthe document according to the logical hierarchy by determining whetherthe document is to be classified into at least one of the second nodeand the first node based on the at least one rule associated with boththe first node and the second node.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings.

FIG. 1 is a network diagram illustrating an example network environmentsuitable for aspects of the present disclosure, according to someexample embodiments.

FIG. 2 is a block diagram showing an example system architecture forperforming aspects of the present disclosure, according to some exampleembodiments

FIG. 3 is a high level diagram showing various examples of types ofhuman communications and what the objectives may be for a naturallanguage model to accomplish, according to some example embodiments.

FIG. 4 is a diagram showing an example flowchart for how different datastructures within the system architecture may be related to one another,according to some example embodiments.

FIG. 5 shows a portion of an example ontology organized in ahierarchical logical structure, and is used to describe an example ofhow aspects of the present disclosure automatically apply rules definedin lower sub-nodes to higher parent nodes where appropriate.

FIG. 6 shows an example interface for associating rules to one nodewithin the example ontology of FIG. 5 , according to some embodiments.

FIG. 7 shows another example screenshot for information about adifferent section of the ontology that does not include the previouslist of rules, according to some embodiments.

FIG. 8 shows additional functionality for discovering topics that may beincluded after the ontology is completed, in part through thepercolation of rules, according to some embodiments.

FIG. 9 shows an example screenshot of additional functionality, wherebythe discover topics functionality may also be indexed by the rulespresent for that particular label, according to some embodiments.

FIG. 10 describes an example methodology for automatically associatingrules applied to a child node in a logical hierarchy to one or moreparent nodes in the hierarchy, according to some embodiments.

FIG. 11 is a block diagram illustrating components of a machine,according to some example embodiments, able to read instructions from amachine-readable medium and perform any one or more of the methodologiesdiscussed herein.

DETAILED DESCRIPTION

Example methods, apparatuses, and systems (e.g., machines) are presentedfor performing natural language processing techniques using humanannotations applied to machine learning techniques of natural language.

Aspects of the present disclosure are presented for the automaticplacement of rules applied to topics (referred to as “labels”) in alogical hierarchy when conducting natural language processing. A goal ofnatural language processing is to classify human communicationsautomatically by machine-based engines. Given the millions and evenbillions of digital communications being generated constantly by humansin the digital age, it is desirable to employ machines to process thecommunications, interpret them and categorize them in meaningful andorganized ways, to make it easier for humans to digest. A naturallanguage model may be trained through a combination of machine learningand rules-based constraints to interpret a specific collection of humancommunications, such as customer service emails for a telecommunicationscompany, or scholarly journals in biology. Examples of generating thesenatural language models are described in application U.S. patentapplication Ser. No. 14/964,517, filed Dec. 9, 2015, and titled “METHODSFOR GENERATING NATURAL LANGUAGE PROCESSING SYSTEMS,” which isincorporated herein by reference.

When generating these natural language models, it is desirable to definea series of rules for the natural language model to follow, that provideinstructions for how the model is to categorize a document based on theappearance of a particular string of characters in the document. Forexample, a rule may specify that the appearance of the word “happy” in adocument may signal to the natural language model that the documentshould be classified as having a “positive sentiment.” In someembodiments, the rules may include a statistical weight, such thatmultiple rules may be applied to the same document and may be combinedto generate a score that overall determines how the document should beclassified. Often, these rules are defined by humans and act as forms of“shortcuts” to help the natural language model classify documents basedon key words or strings of characters.

In some cases, natural language models utilize the rules to classify thehuman communications by following a logical hierarchy of topics orcategories, which may be referred to as an “ontology.” The ontology maybe organized in various logical hierarchies, such as a nested treestructure having multiple nodes and sub-nodes (and subsequent sub-levelsof sub-nodes underneath ad nausea). Further details of ontologies willbe described in more detail below. Each node or sub node in an ontologymay be associated with a set of rules that help the natural languagemodel to determine whether a document should be classified into thatnode.

However, the process for defining rules for each node in the ontologymay be quite burdensome and difficult, particularly if each of the rulesmust be defined manually by one or more human analysts. In addition, itmay be desirable to apply the same rule in a sub node to its parentnode, if logically it makes sense for the rule in the sub node to alsohelp classify a document as fitting into the parent node. It may bedesirable therefore to provide more automatic and efficient ways todefine rules for all of the nodes in an ontology that may be used by anatural language model for natural language processing.

Aspects of the present disclosure describe systems and methods forpercolating rules defined at lower sub nodes in an ontology up to alsobe applied to higher nodes in the ontology in a logical and automaticfashion. The disclosures herein may allow for the generation of naturallanguage models to be quicker and more efficient, as conventionalmethods for generating accurate and reliable natural language modelstend to take weeks to months. The disclosures herein may help reduce theamount of manual inputs made to defining rules for all levels in theontology.

Examples merely demonstrate possible variations. Unless explicitlystated otherwise, components and functions are optional and may becombined or subdivided, and operations may vary in sequence or becombined or subdivided. In the following description, for purposes ofexplanation, numerous specific details are set forth to provide athorough understanding of example embodiments. It will be evident to oneskilled in the art, however, that the present subject matter may bepracticed without these specific details.

Referring to FIG. 1 , a network diagram illustrating an example networkenvironment 100 suitable for performing aspects of the presentdisclosure is shown, according to some example embodiments. The examplenetwork environment 100 includes a server machine 110, a database 115, afirst device 120 for a first user 122, and a second device 130 for asecond user 132, all communicatively coupled to each other via a network190. The server machine 110 may form all or part of a network-basedsystem 105 (e.g., a cloud-based server system configured to provide oneor more services to the first and second devices 120 and 130). Theserver machine 110, the first device 120, and the second device 130 mayeach be implemented in a computer system, in whole or in part, asdescribed below with respect to FIG. 11 . The network-based system 105may be an example of a natural language platform configured to generatenatural language models and perform the various functions as describedherein. The server machine 110 and the database 115 may be components ofthe natural language platform configured to perform these functions.While the server machine 110 is represented as just a single machine andthe database 115 is represented as just a single database, in someembodiments, multiple server machines and multiple databasescommunicatively coupled in parallel or in serial may be utilized, andembodiments are not so limited.

Also shown in FIG. 1 are a first user 122 and a second user 132. One orboth of the first and second users 122 and 132 may be a human user, amachine user (e.g., a computer configured by a software program tointeract with the first device 120), or any suitable combination thereof(e.g., a human assisted by a machine or a machine supervised by ahuman). The first user 122 may be associated with the first device 120and may be a user of the first device 120. For example, the first device120 may be a desktop computer, a vehicle computer, a tablet computer, anavigational device, a portable media device, a smartphone, or awearable device (e.g., a smart watch or smart glasses) belonging to thefirst user 122. Likewise, the second user 132 may be associated with thesecond device 130. As an example, the second device 130 may be a desktopcomputer, a vehicle computer, a tablet computer, a navigational device,a portable media device, a smartphone, or a wearable device (e.g., asmart watch or smart glasses) belonging to the second user 132. Thefirst user 122 and a second user 132 may be examples of users orcustomers interfacing with the network-based system 105 to utilize anatural language model according to their specific needs and forproviding inputs that the natural language platform utilizes forperforming the various functions described herein. In other cases, theusers 122 and 132 may be examples of annotators who are supplyingannotations to documents to be used for training purposes whendeveloping a natural language model. In other cases, the users 122 and132 may be examples of analysts who are providing inputs to the naturallanguage platform to more efficiently train the natural language model.The users 122 and 132 may interface with the network-based system 105through the devices 120 and 130, respectively.

Any of the machines, databases 115, or first or second devices 120 or130 shown in FIG. 1 may be implemented in a general-purpose computermodified (e.g., configured or programmed) by software (e.g., one or moresoftware modules) to be a special-purpose computer to perform one ormore of the functions described herein for that machine, database 115,or first or second device 120 or 130. For example, a computer systemable to implement any one or more of the methodologies described hereinis discussed below with respect to FIG. 11 . As used herein, a“database” may refer to a data storage resource and may store datastructured as a text file, a table, a spreadsheet, a relational database(e.g., an object-relational database), a triple store, a hierarchicaldata store, any other suitable means for organizing and storing data orany suitable combination thereof. Moreover, any two or more of themachines, databases, or devices illustrated in FIG. 1 may be combinedinto a single machine, and the functions described herein for any singlemachine, database, or device may be subdivided among multiple machines,databases, or devices.

The network 190 may be any network that enables communication between oramong machines, databases 115, and devices (e.g., the server machine 110and the first device 120). Accordingly, the network 190 may be a wirednetwork, a wireless network (e.g., a mobile or cellular network), or anysuitable combination thereof. The network 190 may include one or moreportions that constitute a private network, a public network (e.g., theInternet), or any suitable combination thereof. Accordingly, the network190 may include, for example, one or more portions that incorporate alocal area network (LAN), a wide area network (WAN), the Internet, amobile telephone network (e.g., a cellular network), a wired telephonenetwork (e.g., a plain old telephone system (POTS) network), a wirelessdata network (e.g., WiFi network or WiMax network), or any suitablecombination thereof. Any one or more portions of the network 190 maycommunicate information via a transmission medium. As used herein,“transmission medium” may refer to any intangible (e.g., transitory)medium that is capable of communicating (e.g., transmitting)instructions for execution by a machine (e.g., by one or more processorsof such a machine), and can include digital or analog communicationsignals or other intangible media to facilitate communication of suchsoftware.

Referring to FIG. 2 , a diagram 200 is presented showing an examplesystem architecture for performing aspects of the present disclosure,according to some example embodiments. The example system architectureaccording to diagram 200 represents various data structures and theirinterrelationships that may comprise a natural language platform, suchas the network-based system 105. These various data structures may beimplemented through a combination of hardware and software, the detailsof which may be apparent to those with skill in the art based on thedescriptions of the various data structures described herein. Forexample, an API module 205 includes one or more API processors, wheremultiple API processors may be connected in parallel. In some exampleembodiments, the repeating boxes in the diagram 200 represent identicalservers or machines, to signify that the system architecture in diagram200 may be scalable to an arbitrary degree. The API module 205 mayrepresent a point of contact for multiple other modules, includes adatabase module 210, a cache module 215, background processes module220, applications module 225, and even an interface for users 235 insome example embodiments. The API module 205 may be configured toreceive or access data from database module 210. The data may includedigital forms of thousands or millions of human communications. Thecache module 215 may store in more accessible memory various informationfrom the database module 210 or from users 235 or other subscribers.Because the database module 210 and cache module 215 show accessibilitythrough API module 205, the API module 205 can also supportauthentication and authorization of the data in these modules. Thebackground module 220 may be configured to perform a number ofbackground processes for aiding natural language processingfunctionality. Various examples of the background processes include amodel training module, a cross validation module, an intelligent queuingmodule, a model prediction module, a topic modeling module, anannotation aggregation module, an annotation validation module, and afeature extraction module. These various modules are described in moredetail below as well as in non-provisional applications U.S. patentapplication Ser. No. 14/964,520, filed Dec. 9, 2015, and titled“OPTIMIZATION TECHNIQUES FOR ARTIFICIAL INTELLIGENCE”; U.S. patentapplication Ser. No. 14/964,522, filed Dec. 9, 2015, and titled“GRAPHICAL SYSTEMS AND METHODS FOR HUMAN-IN-THE-LOOP MACHINEINTELLIGENCE”; U.S. patent application Ser. No. 14/964,510, filed Dec.9, 2015, and titled “METHODS AND SYSTEMS FOR IMPROVING MACHINE LEARNINGPERFORMANCE”; U.S. patent application Ser. No. 14/964,512, filed Dec. 9,2015, and titled “AN INTELLIGENT SYSTEM THAT DYNAMICALLY IMPROVES ITSKNOWLEDGE AND CODE-BASE FOR NATURAL LANGUAGE UNDERSTANDING”; U.S. patentapplication Ser. No. 14/964,525, filed Dec. 9, 2015, and titled “METHODSAND SYSTEMS FOR LANGUAGE-AGNOSTIC MACHINE LEARNING IN NATURAL LANGUAGEPROCESSING USING FEATURE EXTRACTION”; and U.S. patent application Ser.No. 14/964,528, filed Dec. 9, 2015, and titled “TECHNIQUES FOR COMBININGHUMAN AND MACHINE LEARNING IN NATURAL LANGUAGE PROCESSING,” each ofwhich are incorporated by reference in their entireties. The API module205 may also be configured to support display and functionality of oneor more applications in applications module 225.

Referring to FIG. 3 , a high level diagram 300 is presented showingvarious examples of types of human communications and what theobjectives may be for a natural language model to accomplish. Here,various sources of data, sometimes referred to as a collection ofdocuments 305, may be obtained and stored in, for example database 115,client data store 155, or database modules 210, and may representdifferent types of human communications, all capable of being analyzedby a natural language model. Examples of the types of documents 305include, but are not limited to, posts in social media, emails or otherwritings for customer feedback, pieces of or whole journalisticarticles, commands spoken or written to electronic devices, transcribedcall center recordings; electronic (instant) messages; corporatecommunications (e.g., SEC 10-k, 10-q); confidential documents andcommunications stored on internal collaboration systems (e.g.,SharePoint, Notes), and pieces of or whole scholarly texts.

In some embodiments, at block 310, it may be desired to classify any ofthe documents 305 into a number of enumerated categories or topics,consistent with some of the descriptions mentioned above. This may bereferred to as performing a document-scope task. For example, a user 130in telecommunications may supply thousands of customer service emailsrelated to services provided by a telecommunications company. The user130 may desire to have a natural language model generated thatclassifies the emails into predetermined categories, such as negativesentiment about their Internet service, positive sentiment about theirInternet service, negative sentiment about their cable service, andpositive sentiment about their cable service. As previously mentioned,these various categories for which a natural language model may classifythe emails into, e.g. “negative” sentiment about “Internet service,”“positive” sentiment about “Internet service,” “negative” sentimentabout “cable service,” etc., may be referred to as “labels.” Based onthese objectives, at block 315, a natural language model may begenerated that is tailored to classify these types of emails into thesetypes of labels. The natural language model may utilize a logicalhierarchy that organizes the labels in a logical manner, referred toherein as an ontology. Example applications of utilizing the ontology bythe natural language model will be described more below.

As another example, in some embodiments, at block 320, it may be desiredto extract specific subsets of text from documents, consistent with someof the descriptions mentioned above. This may be another example ofperforming a span-scope task, in reference to the fact that thisfunction focuses on a subset within each document (as previouslymentioned, referred to herein as a “span”). For example, a user 130 maydesire to identify all instances of a keyword, key phrase, or generalsubject matter within a novel. Certainly, this span scope task may beapplied to multiple novels or other documents. Here too, based on thisobjective, at block 315, a natural language model may be generated thatis tailored to perform this function for a specified number ofdocuments.

As another example, in some embodiments, at block 325, it may be desiredto discover what categories the documents may be thematically ortopically organized into in the first place, consistent withdescriptions above about topic modeling. In some cases, the user 130 mayutilize the natural language platform only to perform topic modeling andto discover what topics are most discussed in a specified collection ofdocuments 305. To this end, the natural language platform may beconfigured to conduct topic modeling analysis at block 330. Topicmodeling is discussed in more detail below, as well as in U.S. patentapplication Ser. No. 14/964,520, filed Dec. 9, 2015, and titled“OPTIMIZATION TECHNIQUES FOR ARTIFICIAL INTELLIGENCE”; U.S. patentapplication Ser. No. 14/964,522, filed Dec. 9, 2015, and titled“GRAPHICAL SYSTEMS AND METHODS FOR HUMAN-IN-THE-LOOP MACHINEINTELLIGENCE”; and U.S. patent application Ser. No. 14/964,528, filedDec. 9, 2015, and titled “TECHNIQUES FOR COMBINING HUMAN AND MACHINELEARNING IN NATURAL LANGUAGE PROCESSING,” each of which are incorporatedherein by reference in their entireties. In some cases, it may bedesired to then generate a natural language model that categorizes thedocuments 305 into these newfound topics. Thus, after performing thetopic modeling analysis 230, in some embodiments, the natural languagemodel may also be generated at block 315.

Referring to FIG. 4 , a diagram 400 is presented showing an exampleflowchart for how different data structures within the systemarchitecture may be related to one another, according to some exampleembodiments. Here, the collections data structure 410 represents a setof documents 435 that in some cases may generally be homogenous. Adocument 435 represents a human communication expressed in a singlediscrete package, such as a single tweet, a webpage, a chapter of abook, a command to a device, or a journal article, or any part thereof.Each collection 410 may have one or more tasks 430 associated with it. Atask 430 may be thought of as a classification scheme. For example, acollection 410 of tweets may be classified by its sentiment, e.g. apositive sentiment or a negative sentiment, where each classificationconstitutes a task 430 about a collection 410. A label 445 refers to aspecific prediction about a specific classification. For example, alabel 445 may be the “positive sentiment” of a human communication, orthe “negative sentiment” of a human communication. In some cases, labels445 can be applied to merely portions of documents 435, such asparagraphs in an article or particular names or places mentioned in adocument 435. For example, a label 445 may be a “positive opinion”expressed about a product mentioned in a human communication, or a“negative opinion” expressed about a product mentioned in a humancommunication. In some example embodiments, a task may be a sub-task ofanother task, allowing for a hierarchy or complex network of tasks. Forexample, if a task has a label of “positive opinion,” there might besub-tasks for types of “positives opinions,” like “intention to purchasethe project,” “positive review,” “recommendation to friend,” and so on,and there may be subtasks that capture other relevant information, suchas “positive features.”

Annotations 440 refer to classifications imputed onto a collection 410or a document 435, often times by human input but may also be added byprogrammatic means, such as interpolating from available metadata,generated by a pre-existing natural language model, and generated by atopic modeling process. As an example, an annotation 440 applies a label445 manually to a document 435. In other cases, annotations 440 areprovided by users 235 from pre-existing data. In other cases,annotations 440 may be derived from human critiques of one or moredocuments 435, where the computer determines what annotation 440 shouldbe placed on a document 435 (or collection 410) based on the humancritique. In other cases, with enough data in a language model,annotations 440 of a collection 410 can be derived from one or morepatterns of pre-existing annotations found in the collection 410 or asimilar collection 410.

In some example embodiments, features 450 refer to a library orcollection of certain key words or groups of words that may be used todetermine whether a task 430 should be associated with a collection 410or document 435. Thus, each task 430 has associated with it one or morefeatures 450 that help define the task 430. In some example embodiments,features 450 can also include a length of words or other linguisticdescriptions about the language structure of a document 435, in order todefine the task 430. For example, classifying a document 435 as being alegal document may be based on determining if the document 435 containsa threshold number of words with particularly long lengths, wordsbelonging to a pre-defined dictionary of legal-terms, or words that arerelated through syntactic structures and semantic relationships. In someexample embodiments, features 450 are defined by code, while in othercases features 450 are discovered by statistical methods. In someexample embodiments, features 450 are treated independently, while inother cases features 450 are networked combinations of simpler featuresthat are used in combination utilizing techniques like “deep-learning.”In some example embodiments, combinations of the methods describedherein may be used to define the features 450, and embodiments are notso limited. One or more processors may be used to identify in a document435 the words found in features data structure 450 to determine whattask should be associated with the document 435.

In some example embodiments, a work unit's data structure 455 specifieswhen humans should be tasked to further examine a document 425. Thus,human annotations may be applied to a document 435 after one or morework units 455 is applied to the document 435. The work units 455 mayspecify how many human annotators should examine the document 435 and inwhat order of documents should document 435 be examined. In some exampleembodiments, work units 455 may also determine what annotations shouldbe reviewed in a particular document 435 and what the optimal userinterface should be for review.

In some example embodiments, the data structures 405, 415, 420 and 425represent data groupings related to user authentication and user accessto data in system architecture. For example, the subscribers block 405may represent users and associated identification information about theusers. The subscribers 405 may have associated API keys 415, which mayrepresent one or more authentication data structures used toauthenticate subscribers and provide access to the collections 410.Groups 420 may represent a grouping of subscribers based on one or morecommon traits, such as subscribers 405 belonging to the same company.Individual users 425 capable of accessing the collections 410 may alsoresult from one or more groups 420. In addition, in some cases, eachgroup 420, user 425, or subscriber 405 may have associated with it amore personalized or customized set of collections 510, documents 435,annotations 440, tasks, 430, features 450, and labels 445, based on thespecific needs of the customer.

Referring to FIG. 5 , illustration 500 shows a portion of an exampleontology organized in a hierarchical logical structure. This exampleontology will be used to describe an example of how aspects of thepresent disclosure automatically apply rules defined in lower sub-nodesto higher parent nodes where appropriate.

In this example ontology, a number of characteristics are worth notingfor purposes of this disclosure. For example, the ontology includes atop level node that includes under it the remaining nested structure ofother categories. Multiple nested levels of sub nodes are present, suchthat a natural language model utilizing this ontology may need totraverse multiple levels from the top down before determining ultimatelyunder what node a document should be classified. Among these multiplelevels are some labels that are repeated at an upper and lower level. Inmany other cases, labels are not repeated. For example, the “Production”label is repeated, with sub-labels of “Costumes/Makeup,” “Set Design,”and “Special Effects” underneath.

Referring to FIG. 6 , illustration 600 shows an example interface forassociating rules to one node within the example ontology of FIG. 5 .The concept of the percolation of rules up the hierarchy of nodes in theontology will be described here.

In the bottom left pane of the example interface 600 is the ontology asshown in FIG. 5 . In the main pane, a list of rules 610 is shown. Here,the list of rules 610 is shown as pertaining to a particular node in theontology, in this case “Special Effects” 630, as shown in the main pane.The “Special Effects” label 630 in the ontology in the bottom left paneis shown under the “Production” label 620. The rules 610 each include aphrase and a statistical weight, as shown. Some of the phrases includenotation indicating that phrase may be included as part of a largerphrase or word, or that the phrase may be broken up into multiple partswith extraneous characters in between. This means that when the naturallanguage model is classifying a document, starting from the top of theontology hierarchy, and reaches the “Production” label 620 that includesthe “Special Effects” sub-node label 630, if the document contains theconditions specified in the phrases shown in the list of rules 610, thenthe document is assigned a score according to the weight associated withthe matching phrase. These weights may then be combined into a compositescore according to a weighting algorithm and compared against athreshold value. If the threshold value is satisfied, then the naturallanguage model determines that said document should be classified underthe “Special Effects” label 630. This classification process maycontinue to see if the document should be further classified under anysub nodes of the “Special Effects” label, although in this case theontology shows that there are no lower nodes. Thus, the analysis forthis particular document may complete.

The rules 610 as shown in the main pane may be defined manually. Aninterface may allow analysts to enter a phrase and assign a weight(e.g., from 0 to 1) to build a list of rules at the node in question. Inaddition, other rules may be imported from outside sources, such as fromclient databases and other pre-existing natural language models.Typically, the rules are defined at the very lowest levels of thehierarchy of the ontology, as it is expected that the documents shouldultimately be classified into at least one of the lowest levels of thehierarchy. In other words, if there exists a sub node label in theontology after the natural language model determines that a documentshould be classified into the parent node, then the natural languagemodel continues classifying the document based on the sub node labels.This would continue until there are there are no lower levels in thehierarchy to traverse. Thus, it would be natural and is typicallyconventional to define rules at the lowest levels.

However, it may also be desirable to define rules at any of the parentnodes in the hierarchical ontology. The rules applied to the parentnodes may be used in a similar analysis for traversing down thehierarchical ontology when classifying a document. At least some ofthese rules may also be defined manually or may be imported, in aprocess similar to that described in FIG. 6 .

In addition, it may be logical to associate the list of rules in thelowest level with parent nodes. For example, it may be logical toassociate the list of rules 610 not only with the lowest sub-node 630 of“Special Effects,” but also with its parent node “Production” 620, andmaybe even with higher parent nodes. This is because the rules definedat a lower level may also logically apply to labels in upper levels. Forexample, the phrases in rules 610 associated with “Special Effects” mayalso describe documents pertaining to “Production,” and therefore thesame rules defined at the lower level should apply to the upper level(e.g., the rule associating “green screen” with Special Effects may alsobe useful for classifying the document as one belonging to Production).In addition, it may also make sense to apply the same rules of “SpecialEffects” to classify a document as belonging to an even higher level.Thus, it may be desirable to have the list 610 apply not only to theSpecial Effects label, but multiple higher levels in the ontologyhierarchy.

Rather than manually copy each of the rules 610 into upper levels, insome embodiments, the natural language platform interface may beconfigured to automatically percolate the rules in lower levels of nodesto be applied to upper levels of nodes appropriately and logically. Forexample, the natural language platform may be configured to identify atag or other criterion in a higher level of the ontology, and based onthat, automatically include the rules in the lower level to said higherlevel. Furthermore, if the same tag or similar criterion is identifiedin the very next higher level of the hierarchy, the list of rules in thelower level may also be included in said next higher level. This maycontinue up the hierarchy of the ontology until the tag or other similarcriterion is no longer found in the higher level nodes.

In this example of illustration 600, the indication to percolate thelist of rules 610 up the logical hierarchy is based on the double namingof a node, such as “Production” 620, as shown. Thus, the rules 610 asoriginally defined in the “Special Effects” label 630 are automaticallycopied into the Production label 620 for use in determining by thenatural language model whether a document should be classified at thislevel. Furthermore, if the next higher level above Production 620 alsoincluded the double naming convention, said next higher level would alsoinclude the list of rules 610 from the Special Effects label 630, suchthat the natural language model will also use these rules in itsanalysis to determine whether a document should be classified at thatlevel.

Other example tags or criteria for indicating to percolate the rules upthe logical hierarchical ontology may include a metadata flag that isturned on or enabled when the ontology is being created, categoricalrules within the ontology itself that define when rules should bepercolated up the hierarchy, and machine learning techniques that learnwhether the inclusion of some rules into higher levels improvesperformance of the natural language model.

Referring to FIG. 7 , illustration 700 shows another example screenshotfor information about a different section of the ontology that does notinclude the previous list of rules. Here, toward the highest levels ofthe logical hierarchy of the ontology, the labels “Relevant” and“Irrelevant” 710 do not include the previous list of rules 610pertaining to “Special Effects.” For example, the main pane 720 does notshow any of the rules 610 included in the list of rules. This is alsodue to the fact that, according to the previously mentioned indicators,the labels “Relevant” and “Irrelevant” are not repeated twice in theontology as shown in the bottom left pane. Furthermore, it may desirableto not percolate the list of rules continually up all levels of thehierarchical ontology because it may not be logical to do so. Forexample, in determining whether a document is “relevant” or “irrelevant”to the ontology in question (here, it is “Film_Review_Demo”), documentscontaining certain words or phrases more specific to “Special Effects”may not be helpful. For example, a document that includes a news reportabout “gas pipe explosion” would not be deemed relevant for thisontology, although “explosion” is a phrase that would ultimately helpclassify the document under “Special Effects,” assuming the document wasfirst deemed to be “Relevant” and was classified further down thesub-levels of that hierarchy. Thus, it makes sense that the percolationof rules be limited to only so many levels, the determination of whichis made possible according to aspects of the present disclosure.

Referring to FIG. 8 , illustration 800 shows additional functionalityfor discovering topics that may be included after the ontology iscompleted, in part through the percolation of rules, according to someembodiments. Once the ontology is fully defined and settled, and afterthe natural language model is trained (for additional details for how anatural language model is trained, refer U.S. patent application Ser.No. 14/964,517, filed Dec. 9, 2015, and titled “METHODS FOR GENERATINGNATURAL LANGUAGE PROCESSING SYSTEMS,” which again is incorporated hereinby reference), the natural language model may classify documentsaccording to the organizational structure of the ontology. Ordinarily,the natural language model may access many documents in a collection andcategorize them into one or more labels (i.e., the nodes in theontology). In some cases, however, it may be desirable by a client oruser to be informed about what kinds of documents are categorized into aparticular label, rather than to have the natural language modelclassify all documents into multiple labels.

Aspects of the present disclosure allow for this specific “discovertopics” functionality, by providing to the client or user only thoseparticular documents that fall into a specified label in the ontology.For example, a user may select in an interface menu of illustration 800the “Discover topics” button 810, when the “Action” label 820 isselected. By selecting this feature, the natural language model may beconfigured to analyze the documents in the collection and provide onlythose documents that traverse through the ontology and ultimately end atthe “Action” label 820, as shown in the example ontology on thebottom-left pane. In this way, some processing may be reduced bylimiting the amount of processing performed on other documents that donot fit into the “Action” label 820. For example, if when traversing thehierarchical ontology, starting from the top node of “Film_Review_Demo,”it is determined that a document falls under the label of “irrelevant,”then the analysis can stop and the next document may be processed. Asanother example, if it is determined that a document falls under thelabel of “relevant” but does not fall under the label of “Genre,” thenthe analysis also can stop and the next document may be processed. Inaddition, this functionality allows a user to specifically identifyexamples of particular labels of interest, and streamlines the amount ofinformation the user has to wait for before accessing that information.

In some embodiments, the processing to allow for this “discover topics”functionality may be aided by the caching of documents alreadyclassified by the natural language model. For example, the discovertopics functionality may be enabled during or after the natural languagemodel has already analyzed a number of documents and classified them.These results may already be stored in a database and may be availablefor display. Any documents already classified under the desired labelmay be readily available for display therefore.

In some embodiments, the discover topics functionality may be availableafter conducting the percolation of rules and solidifying the structureof the ontology in general. In other cases, the percolation of rules upthe hierarchy may not be necessary in order to solidify the ontologybefore utilizing the discover topics functionality.

Referring to FIG. 9 , illustration 900 shows an example screenshot ofadditional functionality, whereby the discover topics functionality mayalso be indexed by the rules present for that particular label,according to some embodiments. The natural language model may beconfigured to attribute a classified document to the factors theresulted in its classification, such as what rules were triggered.Metadata may be appended to the document that records thesedeterminations once classified and may be stored in a database. In thisway, for example, each rule in the rules list 910 may be selected duringa discover topics function, and documents classified according to saidrule may be presented in the display. In this example of FIG. 9 , therules list 910 is associated with the “Action” label 920. Each of therules may be used to index the documents, whereby clicking on a rule maythen cause display of all of the documents that were classified at leastin part by that rule. In some cases, the documents may appear undermultiple rules, since more than one rule may have contributed to thedocument's classification. In other examples, the documents may bedisplayed and the phrases of each rule used to classify said documentmay be indicated with the document. In this way, the interface may showall of the different rules triggered to classify the document.

Referring to FIG. 10 , flowchart 1000 describes an example methodologyfor automatically associating rules applied to a child node in a logicalhierarchy to one or more parent nodes in the hierarchy, according tosome embodiments. The example description herein may be consistent withthe various descriptions in the previous figures. The examplemethodology may be performed by a natural language platform configuredto generate a natural language model and utilize the natural languagemodel to classify a collection of documents, such as the network-basedsystem 105 and the more detailed system architecture 200.

At block 1005, the natural language platform may be configured to accessa rule associated with a child node in a logical hierarchy, alsoreferred to herein as an ontology. The rule may help define a factor orcriterion for determining whether a document should be classified intothe child node. An example factor or criterion may include applying aweight to the child node if the phrase associated with the rule is foundin the document being classified. Examples and example properties of thelogical hierarchy may include the various descriptions of variousontologies described herein. The natural language platform may accessthe rule based on a user defining the rule in a user interfaceassociated with the child node. It is noted that the rule may first beassociated with the child node before being associated with any othernode in the ontology.

At block 1010, the natural language platform may be configured toidentify a percolation criterion associated with a parent node in theontology indicating that the rule associated with the child node is alsoto be associated with the parent node. An example of a percolationcriterion is that the name of the parent node is repeated at differentlevels within the ontology. Examples of this type of percolationcriterion are described in the previous figures. Another exampleincludes the inclusion of a metadata tag associated with the parentsnode that signals that all rules in the lower children nodes should bealso associated with the parent node. In some cases, these percolationcriteria may be specified by the user and inputted into a user interfaceof the natural language platform. In other cases, the natural languageplatform may employ machine learning methods to learn when rules in thechildren nodes should be associated with the parent nodes.

At block 1015, the natural language platform may be configured toassociate the rule originally with the child node to be with the parentsnode based on satisfying or identifying the percolation criterion.Intermediate operations may include automatically copying the rule fromthe child node and duplicating it into the parent node, and also makingthe rule displayable when the user surveys information about the parentnode.

In some embodiments, this process of percolating a rule up higher levelsin the logical hierarchy may continue for subsequent parent nodes forall subsequent parent nodes that satisfy the percolation criterion or atleast have the percolation criterion present. Thus, the examplemethodology may cycle back to block 1010 from block 1015, and may repeatthe process until a percolation criterion associated with the nexthigher parents node is no longer met or is not present. An example ofthe rule in the lowest child node not percolating up the entire logicalhierarchy is described in FIG. 7 .

In general, this example methodology may be applied for every rule inevery child node of the logical hierarchy. Thus, after this process iscompleted for each and all rules that may be defined, at block 1020, thenatural language model may be configured to classify documents accordingto the ontology and based on utilizing the at least one rule describedin the present methodology that is associated with both the child nodeand the parent node. Of course, said rule may also be associated withhigher parent nodes, and embodiments are not so limited. In addition,other rules may also be afforded the same treatment, and the presentmethodology is not limited to just a single rule.

In some embodiments, the process of applying a percolated rule to adocument involves identifying the regions of text within the documentthat match the criteria specified by the rule. Because percolated rulesmay need to be applied to multiple nodes in the ontology, and becauserule application may be computationally intensive, it is desirable tostore these matching regions in a cache associated with the document andthe rule to avoid the full rule application computation for each suchpercolated node. In some embodiments, this cache may use a hash functionof the document content and metadata, such as the Secure Hash Algorithm2 (SHA-2), to identify such cached documents. The matching regions maybe stored in any suitable memory, including the API server's localmemory, a distributed in-memory caching server such as Redis ormemcached, or in a persistent database.

In some embodiments, the methodology may also include employing thediscover topics functionality described in FIG. 8 . This functionalitymay be employed after the ontology is well defined based at least inpart on any percolation of rules where applicable. In some embodiments,the discover topics functionality may be indexed according to thevarious rules associated with the label in question. An example ofindexing according to the rules as described in FIG. 9 .

Referring to FIG. 11 , the block diagram illustrates components of amachine 1100, according to some example embodiments, able to readinstructions 1124 from a machine-readable medium 1122 (e.g., anon-transitory machine-readable medium, a machine-readable storagemedium, a computer-readable storage medium, or any suitable combinationthereof) and perform any one or more of the methodologies discussedherein, in whole or in part. Specifically, FIG. 11 shows the machine1100 in the example form of a computer system (e.g., a computer) withinwhich the instructions 1124 (e.g., software, a program, an application,an applet, an app, or other executable code) for causing the machine1100 to perform any one or more of the methodologies discussed hereinmay be executed, in whole or in part.

In alternative embodiments, the machine 1100 operates as a standalonedevice or may be connected (e.g., networked) to other machines. In anetworked deployment, the machine 1100 may operate in the capacity of aserver machine 110 or a client machine in a server-client networkenvironment, or as a peer machine in a distributed (e.g., peer-to-peer)network environment. The machine 1100 may include hardware, software, orcombinations thereof, and may, as example, be a server computer, aclient computer, a personal computer (PC), a tablet computer, a laptopcomputer, a netbook, a cellular telephone, a smartphone, a set-top box(STB), a personal digital assistant (PDA), a web appliance, a networkrouter, a network switch, a network bridge, or any machine capable ofexecuting the instructions 1124, sequentially or otherwise, that specifyactions to be taken by that machine. Further, while only a singlemachine 1100 is illustrated, the term “machine” shall also be taken toinclude any collection of machines that individually or jointly executethe instructions 1124 to perform all or part of any one or more of themethodologies discussed herein.

The machine 1100 includes a processor 1102 (e.g., a central processingunit (CPU), a graphics processing unit (GPU), a digital signal processor(DSP), an application specific integrated circuit (ASIC), aradio-frequency integrated circuit (RFIC), or any suitable combinationthereof), a main memory 1104, and a static memory 1106, which areconfigured to communicate with each other via a bus 1108. The processor1102 may contain microcircuits that are configurable, temporarily orpermanently, by some or all of the instructions 1124 such that theprocessor 1102 is configurable to perform any one or more of themethodologies described herein, in whole or in part. For example, a setof one or more microcircuits of the processor 1102 may be configurableto execute one or more modules (e.g., software modules) describedherein.

The machine 1100 may further include a video display 1110 (e.g., aplasma display panel (PDP), a light emitting diode (LED) display, aliquid crystal display (LCD), a projector, a cathode ray tube (CRT), orany other display capable of displaying graphics or video). The machine1100 may also include an alphanumeric input device 1112 (e.g., akeyboard or keypad), a cursor control device 1114 (e.g., a mouse, atouchpad, a trackball, a joystick, a motion sensor, an eye trackingdevice, or other pointing instrument), a storage unit 1116, a signalgeneration device 1118 (e.g., a sound card, an amplifier, a speaker, aheadphone jack, or any suitable combination thereof), and a networkinterface device 1120.

The storage unit 1116 includes the machine-readable medium 1122 (e.g., atangible and non-transitory machine-readable storage medium) on whichare stored the instructions 1124 embodying any one or more of themethodologies or functions described herein, including, for example, anyof the descriptions of FIGS. 1-10 . The instructions 1124 may alsoreside, completely or at least partially, within the main memory 1104,within the processor 1102 (e.g., within the processor's cache memory),or both, before or during execution thereof by the machine 1100. Theinstructions 1124 may also reside in the static memory 1106.

Accordingly, the main memory 1104 and the processor 1102 may beconsidered machine-readable media 1122 (e.g., tangible andnon-transitory machine-readable media). The instructions 1124 may betransmitted or received over a network 1126 via the network interfacedevice 1120. For example, the network interface device 1120 maycommunicate the instructions 1124 using any one or more transferprotocols (e.g., HTTP). The machine 1100 may also represent examplemeans for performing any of the functions described herein, includingthe processes described in FIGS. 1-10 .

In some example embodiments, the machine 1100 may be a portablecomputing device, such as a smart phone or tablet computer, and have oneor more additional input components (e.g., sensors or gauges) (notshown). Examples of such input components include an image inputcomponent (e.g., one or more cameras), an audio input component (e.g., amicrophone), a direction input component (e.g., a compass), a locationinput component (e.g., a GPS receiver), an orientation component (e.g.,a gyroscope), a motion detection component (e.g., one or moreaccelerometers), an altitude detection component (e.g., an altimeter),and a gas detection component (e.g., a gas sensor). Inputs harvested byany one or more of these input components may be accessible andavailable for use by any of the modules described herein.

As used herein, the term “memory” refers to a machine-readable medium1122 able to store data temporarily or permanently and may be taken toinclude, but not be limited to, random-access memory (RAM), read-onlymemory (ROM), buffer memory, flash memory, and cache memory. While themachine-readable medium 1122 is shown in an example embodiment to be asingle medium, the term “machine-readable medium” should be taken toinclude a single medium or multiple media (e.g., a centralized ordistributed database 115, or associated caches and servers) able tostore instructions 1124. The term “machine-readable medium” shall alsobe taken to include any medium, or combination of multiple media, thatis capable of storing the instructions 1124 for execution by the machine1100, such that the instructions 1124, when executed by one or moreprocessors of the machine 1100 (e.g., processor 1102), cause the machine1100 to perform any one or more of the methodologies described herein,in whole or in part. Accordingly, a “machine-readable medium” refers toa single storage apparatus or device 120 or 130, as well as cloud-basedstorage systems or storage networks that include multiple storageapparatus or devices 120 or 130. The term “machine-readable medium”shall accordingly be taken to include, but not be limited to, one ormore tangible (e.g., non-transitory) data repositories in the form of asolid-state memory, an optical medium, a magnetic medium, or anysuitable combination thereof.

Furthermore, the machine-readable medium 1122 is non-transitory in thatit does not embody a propagating signal. However, labeling the tangiblemachine-readable medium 1122 as “non-transitory” should not be construedto mean that the medium is incapable of movement; the medium should beconsidered as being transportable from one physical location to another.Additionally, since the machine-readable medium 1122 is tangible, themedium may be considered to be a machine-readable device.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute softwaremodules (e.g., code stored or otherwise embodied on a machine-readablemedium 1122 or in a transmission medium), hardware modules, or anysuitable combination thereof. A “hardware module” is a tangible (e.g.,non-transitory) unit capable of performing certain operations and may beconfigured or arranged in a certain physical manner. In various exampleembodiments, one or more computer systems (e.g., a standalone computersystem, a client computer system, or a server computer system) or one ormore hardware modules of a computer system (e.g., a processor 1102 or agroup of processors 1102) may be configured by software (e.g., anapplication or application portion) as a hardware module that operatesto perform certain operations as described herein.

In some embodiments, a hardware module may be implemented mechanically,electronically, or any suitable combination thereof. For example, ahardware module may include dedicated circuitry or logic that ispermanently configured to perform certain operations. For example, ahardware module may be a special-purpose processor, such as a fieldprogrammable gate array (FPGA) or an ASIC. A hardware module may alsoinclude programmable logic or circuitry that is temporarily configuredby software to perform certain operations. For example, a hardwaremodule may include software encompassed within a general-purposeprocessor 1102 or other programmable processor 1102. It will beappreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multiplehardware modules exist contemporaneously, communications may be achievedthrough signal transmission (e.g., over appropriate circuits and buses1108) between or among two or more of the hardware modules. Inembodiments in which multiple hardware modules are configured orinstantiated at different times, communications between such hardwaremodules may be achieved, for example, through the storage and retrievalof information in memory structures to which the multiple hardwaremodules have access. For example, one hardware module may perform anoperation and store the output of that operation in a memory device towhich it is communicatively coupled. A further hardware module may then,at a later time, access the memory device to retrieve and process thestored output. Hardware modules may also initiate communications withinput or output devices, and can operate on a resource (e.g., acollection of information).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors 1102 that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors 1102 may constitute processor-implementedmodules that operate to perform one or more operations or functionsdescribed herein. As used herein, “processor-implemented module” refersto a hardware module implemented using one or more processors 1102.

Similarly, the methods described herein may be at least partiallyprocessor-implemented, a processor 1102 being an example of hardware.For example, at least some of the operations of a method may beperformed by one or more processors 1102 or processor-implementedmodules. As used herein, “processor-implemented module” refers to ahardware module in which the hardware includes one or more processors1102. Moreover, the one or more processors 1102 may also operate tosupport performance of the relevant operations in a “cloud computing”environment or as a “software as a service” (SaaS). For example, atleast some of the operations may be performed by a group of computers(as examples of machines 1100 including processors 1102), with theseoperations being accessible via a network 1126 (e.g., the Internet) andvia one or more appropriate interfaces (e.g., an API).

The performance of certain operations may be distributed among the oneor more processors 1102, not only residing within a single machine 1100,but deployed across a number of machines 1100. In some exampleembodiments, the one or more processors 1102 or processor-implementedmodules may be located in a single geographic location (e.g., within ahome environment, an office environment, or a server farm). In otherexample embodiments, the one or more processors 1102 orprocessor-implemented modules may be distributed across a number ofgeographic locations.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine 1100 (e.g., a computer) that manipulates ortransforms data represented as physical (e.g., electronic, magnetic, oroptical) quantities within one or more memories (e.g., volatile memory,non-volatile memory, or any suitable combination thereof), registers, orother machine components that receive, store, transmit, or displayinformation. Furthermore, unless specifically stated otherwise, theterms “a” or “an” are herein used, as is common in patent documents, toinclude one or more than one instance. Finally, as used herein, theconjunction “or” refers to a non-exclusive “or,” unless specificallystated otherwise.

The present disclosure is illustrative and not limiting. Furthermodifications will be apparent to one skilled in the art in light ofthis disclosure and are intended to fall within the scope of theappended claims.

What is claimed is:
 1. A method for improving natural languageprocessing conducted by a natural language model, the method comprising:accessing, at a first node in a logical hierarchy configured to guideclassification of a plurality of documents by the natural languagemodel, at least one rule associated with the first node, said at leastone rule defining a first factor for determining whether a documentamong the plurality of documents is to be classified into the firstnode; identifying a percolation criterion associated with a second nodein the logical hierarchy that is a parent node to the first node, saidpercolation criterion indicating that the at least one rule associatedwith the first node is to be associated also with the second node; andbased on the identified percolation criterion, associating the at leastone rule with the second node such that the at least one rule defines asecond factor for determining whether the document is to also beclassified into the second node, the document being classifiableaccording to the logical hierarchy into at least one of the second nodeand the first node based on the at least one rule associated with boththe first node and the second node.
 2. The method of claim 1, whereinthe percolation criterion is a first percolation criterion, and themethod further comprises: identifying a second percolation criterionassociated with a third node in the logical hierarchy that is a parentnode to the second node, said percolation criterion indicating that theat least one rule associated with the first node is to be associatedalso with the third node; and based on the identified second percolationcriterion, associating the at least one rule with the third node suchthat the at least one rule defines a third factor for determiningwhether the document is to also be classified into the third node. 3.The method of claim 1, further comprising: determining that a secondpercolation criterion is not associated with a third node in the logicalhierarchy that is a parent node to the second node; and based on saiddetermination, limiting the at least one rule to be associated only tothe first node and the second node.
 4. The method of claim 1, whereinthe percolation criterion includes the second node being repeated atleast once within the logical hierarchy.
 5. The method of claim 1,wherein the percolation criterion includes a metadata tag that isenabled.
 6. The method of claim 1, wherein the percolation criterion isspecified by user input to a user interface.
 7. The method of claim 1,wherein said accessing is based on the at least one rule being definedby user input to a user interface associated with the first node.
 8. Asystem comprising: a memory; and a processor coupled to the memory andconfigured to: access, at a first node in a logical hierarchy configuredto guide classification of a plurality of documents by the naturallanguage model, at least one rule associated with the first node, saidat least one rule defining a first factor for determining whether adocument among the plurality of documents is to be classified into thefirst node; identify a percolation criterion associated with a secondnode in the logical hierarchy that is a parent node to the first node,said percolation criterion indicating that the at least one ruleassociated with the first node is to be associated also with the secondnode; and based on the identified percolation criterion, associate theat least one rule with the second node such that the at least one ruledefines a second factor for determining whether the document is to alsobe classified into the second node, the document being classifiableaccording to the logical hierarchy into at least one of the second nodeand the first node based on the at least one rule associated with boththe first node and the second node.
 9. The system of claim 8, whereinthe percolation criterion is a first percolation criterion, and theprocessor is further configured to: identify a second percolationcriterion associated with a third node in the logical hierarchy that isa parent node to the second node, said percolation criterion indicatingthat the at least one rule associated with the first node is to beassociated also with the third node; and based on the identified secondpercolation criterion, associate the at least one rule with the thirdnode such that the at least one rule defines a third factor fordetermining whether the document is to also be classified into the thirdnode.
 10. The system of claim 8, wherein the processor is furtherconfigured to: determine that a second percolation criterion is notassociated with a third node in the logical hierarchy that is a parentnode to the second node; and based on said determination, limit the atleast one rule to be associated only to the first node and the secondnode.
 11. The system of claim 8, wherein the percolation criterionincludes the second node being repeated at least once within the logicalhierarchy.
 12. The system of claim 8, wherein the percolation criterionincludes a metadata tag that is enabled.
 13. The system of claim 8,wherein the percolation criterion is specified by user input to a userinterface.
 14. The system of claim 8, wherein said accessing is based onthe at least one rule being defined by user input to a user interfaceassociated with the first node.
 15. A non-transitory computer readablemedium comprising instructions that, when executed by a processor, causethe processor to perform operations comprising: accessing, at a firstnode in a logical hierarchy configured to guide classification of aplurality of documents by the natural language model, at least one ruleassociated with the first node, said at least one rule defining a firstfactor for determining whether a document among the plurality ofdocuments is to be classified into the first node; identifying apercolation criterion associated with a second node in the logicalhierarchy that is a parent node to the first node, said percolationcriterion indicating that the at least one rule associated with thefirst node is to be associated also with the second node; and based onthe identified percolation criterion, associating the at least one rulewith the second node such that the at least one rule defines a secondfactor for determining whether the document is to also be classifiedinto the second node; the document being classifiable according to thelogical hierarchy into at least one of the second node and the firstnode based on the at least one rule associated with both the first nodeand the second node.
 16. The computer readable medium of claim 15,wherein the percolation criterion is a first percolation criterion, andthe operations further comprise: identifying a second percolationcriterion associated with a third node in the logical hierarchy that isa parent node to the second node, said percolation criterion indicatingthat the at least one rule associated with the first node is to beassociated also with the third node; and based on the identified secondpercolation criterion, associating the at least one rule with the thirdnode such that the at least one rule defines a third factor fordetermining whether the document is to also be classified into the thirdnode.
 17. The computer readable medium of claim 15, wherein theoperations further comprise: determining that a second percolationcriterion is not associated with a third node in the logical hierarchythat is a parent node to the second node; and based on saiddetermination, limiting the at least one rule to be associated only tothe first node and the second node.
 18. The computer readable medium ofclaim 15, wherein the percolation criterion includes the second nodebeing repeated at least once within the logical hierarchy.
 19. Thecomputer readable medium of claim 15, wherein the percolation criterionincludes a metadata tag that is enabled.
 20. The computer readablemedium of claim 15, wherein the percolation criterion is specified byuser input to a user interface.